Add ephemeral EC2 Harbor backend - #1131

What changed

• Adds opt-in environment=ec2 execution with one public-IP, key-only SSH EC2 instance per Harbor trial.
• Keeps Daytona as the automatic CPU default. EC2 rejects accelerators, attach/retain mode, private networking, and caller overrides of platform-owned launch settings.
• Gives EC2 workers dedicated AWS-control and SSH secrets; the reconciler receives control credentials only, while the API and dispatcher receive neither.
• Materializes AWS credentials and the SSH key in mode-0600 temporary files for the owning operation, then removes them.
• Tags instances and root volumes with protected account, deployment, trial, worker-job, Harbor-session, and task ownership metadata.
• Persists an ec2://<account>/<region>/<instance> handle immediately after launch and requires both protected tags and the sandbox ledger for teardown.
• Handles normal completion, cancellation, stale-heartbeat cleanup, and scheduled orphan reconciliation, with a 30-minute grace and a 14-hour hard maximum age.
• Adds a provider-wide EC2 capacity lease independent of model queue slots. Cancellation retains the lease until reconciliation proves the owner terminal, so replacement workers cannot temporarily exceed the cap.
• Supports pinned in-process Harbor and override-Harbor child execution with identical resolved configuration and tags.
• Routes the EC2 lane through Modal, in-process, Docker, and Kubernetes dispatchers.
• Pins Harbor to a7caa586de620f8a7103018f99ce9e1603f90eb4, including lifecycle identity and structured environment-subphase timing.
• Installs Harbor's EC2 dependencies and OpenSSH in standalone and Modal worker images.
• Documents IAM, networking, Modal secrets, CLI use, rollout, and canary requirements.

No ECS cluster, Auto Scaling group, or AWS infrastructure provisioning is introduced. Three Oddish migrations add execution-lane routing, sandbox ownership/lifecycle records, and EC2 capacity leases.

Companion Harbor PR: abundant-ai/harbor#23.

Safety and cleanup

• EC2 credentials are reference-counted behind one backend-owned lease so concurrent in-process runs cannot delete each other's temporary files.
• API cancellation delegates to a dedicated control-secret-bearing Modal function, which resolves the backend directly and cannot recurse through the API delegate.
• Reconciliation snapshots AWS before opening the shared DB transaction and performs termination only after commit.
• Discovery and teardown verify protected account/deployment tags and the sandbox ledger.
• AWS and teardown failures are logged and metered instead of silently skipped.
• Disposable PR previews intentionally do not enable EC2, preventing preview shutdown/redeploy from orphaning deployment-scoped instances.
• The default provider-wide cap is 16 instances per Oddish deployment; each active EC2 trial consumes one lease.

Validation

• Existing EC2/Harbor/runner/cleanup/policy suites passed during development; the only selected integration failure required a local PostgreSQL service that was not running.
• Final cancellation-capacity patch: 71 focused backend tests passed.
• Ruff, Black, and git diff --check passed on the final patch.
• The previous PR head passed migration guards/upgrades, CLI smoke, dashboard E2E, preview deployment, Vercel, and Bugbot. Final-head CI is tracked in the PR checks.
• Companion Harbor PR Add experiment-level cost tracking with token-based estimation #23 is mergeable and clean; Linux, Windows, lint, and type checks are green.

AWS canary evidence

Task: 13-P1-distractor-volume-shell-6f90339c

NOP and oracle baseline — experiment ce649b18

Both trials used the pinned Harbor SHA and completed with no inner Harbor exception. All 18 setup and teardown subphases succeeded.

The NOP used the shortened verifier_finalized_without_declaration path; its long verifier duration is the task's evidence floor, not the one-hour fallback.

3× Fireworks GLM-5p2 — experiment 3ac1d042

All three trials launched an EC2 instance, reached the native k3s/Helm task environment, ran the verifier, uploaded artifacts, completed all 18 lifecycle subphases successfully, and terminated the instance. Exact-ID AWS inspection then proved every root volume deleted and active_managed_volumes=[].

Oddish currently reports these three rows as SUCCESS because the verifier completed, while Harbor's inner results preserve the agent exceptions. The EC2 backend E2E and destructive cleanup paths are proven; these runs do not establish Fireworks model reliability or task-solving success.

Performance

The measured cold path is dominated by two waits:

• EC2 running/status/IP/SSH: approximately 101–162 seconds in the final matrix.
• Task Helm install: approximately 346–388 seconds.

The prebuilt 33 GB workload-image AMI was not retained because lazy EBS snapshot initialization made it slower. Daytona remains the hosted CPU default. Follow-up optimization should focus on task chart readiness and a small tools-only AMI rather than baking the workload image store without Fast Snapshot Restore or explicit volume initialization.

EC2 remains opt-in after merge.